Calculating kappa measures of agreement and standard errors using SAS software: some tricks and traps

SAS/STAT® procedure FREQ is the place to start when you need to compute measures of rater or test agreement on the classic kappa scale (Cohen 1960), namely, the ratio of the actual improvement over chance to the maximum possible improvement over chance. But when you see the frustrating message "WARNING: AGREE statistics are computed only for tables where the number of rows equals the number of columns" and you know your kappa statistic is computable, where do you go next? If you need an estimate of the standard error of your kappa statistic when your data are not a simple random sample from your population, how can you use SAS software to compute it? How best to display your estimates and standard errors graphically? This paper provides some answers, a sample graphical summary of test sensitivity and specificity, and a macro useful for stratified samples. Examples are drawn from other kappa papers and the cardiovascular testing literature. INTRODUCTION Given a population, a pair of assessments X and Y, and a measure of agreement between X andY, 2(X,Y), it maybe useful to convert this measure to a standard scale where l represents perfect agreement and 0 represents chance agreement, i.e., 2(X,Y) for a population in which X andY are statistically independent. In honor of Jacob Cohen, who proposed the formula below: